Image processing apparatus for endoscope and endoscope system

ABSTRACT

An image processing apparatus for an endoscope includes an image acquisition circuit configured to acquire an image captured by using the endoscope, an interference area setting circuit configured to set, on the image, an interference area having a different characteristic from an observation target in the image, and an interference area corrector configured to perform correction on the image, the correction including a reduction in luminance of the interference area. The interference area setting circuit includes an interference candidate area extractor configured to extract at least one interference candidate area based on the different characteristic from the observation target, and an interference area selector configured to select the interference area from the at least one interference candidate area based on luminance distribution in the at least one interference candidate area.

This is a Continuation Application of PCT Application No.PCT/JP2017/006645, filed Feb. 22, 2017, the entire contents of which areincorporated herein by reference.

BACKGROUND

The present invention relates to an image processing apparatus for anendoscope and an endoscope system.

A procedure may be performed using various procedure tools underobservation with an endoscope. In such a procedure, when a distalportion of the endoscope which captures an image and a procedure toolthat performs the procedure are positioned at a close distance, a rangeof a capturing view field may be blocked by the procedure tool.Furthermore, illumination light for the capturing may be intensivelyreflected from the procedure tool, and luminance of the procedure toolmay significantly increase on the image. An area with high-luminance islikely to notice, and thus user's eyes are likely to look at thehigh-luminance area.

In recent years, a three-dimensional endoscope using binocular vision isused in some cases. A subject image acquired by the three-dimensionalendoscope is displayed by using a three-dimensional stereoscopic displayapparatus such as a three-dimensional (3D) display. When the distalportion of the three-dimensional endoscope and the procedure tool thatperforms the procedure are positioned at a close distance duringcapturing by the endoscope, it is not possible to achieve a stereoscopicview of an image displayed by the three-dimensional stereoscopic displayapparatus in some cases. In this case, a user feels discomfort. Hence,when the distal portion of the endoscope and the procedure tool thatperforms the procedure are positioned at a close distance, an area thatmakes the user feel discomfort is likely to catch the user's eyes. Inaddition, an image area contained only one of a pair of right and leftimages which are acquired to obtain disparity can be produced. A space,in which the three-dimensional endoscope is used, has a lumen shape inmany cases, a subject that is present in the image area contained inonly one of the pair of right and left images has a close distance tothe distal portion of the endoscope in many cases. Therefore, ahigh-luminance area is likely to be produced. Further, the subject iscontained in only one of the pair of right and left images, and thus itis not possible to achieve the stereoscopic view of the subject. Theuser who views the image acquired by the three-dimensional endoscope mayfeel discomfort.

For example, U.S. Patent Application Publication No. 2014/0088353discloses a technology related to a reduction in discomfort that a usermay feel when using a three-dimensional endoscope. In this technology,an image, in which a procedure tool is captured, and an image, in whichthe procedure tool is not captured, are acquired. An area, in which theprocedure tool is captured, is detected in the image in which theprocedure tool is captured, and the image within the area is replacedwith an image of an area corresponding to the area in the image in whichthe procedure tool is not captured. A correspondence of the areas inboth images can be acquired by performing area-based matching of bothimages.

SUMMARY

According to an exemplary embodiment, an image processing apparatus foran endoscope includes an image acquisition circuit configured to acquirean image captured by using the endoscope, an interference area settingcircuit configured to set, on the image, an interference area having adifferent characteristic from an observation target in the image, and aninterference area corrector configured to perform correction on theimage, the correction including a reduction in luminance of theinterference area, wherein the interference area setting circuitincludes an interference candidate area extractor configured to extractat least one interference candidate area based on the differentcharacteristic from the observation target, and an interference areaselector configured to select the interference area from the at leastone interference candidate area based on luminance distribution in theat least one interference candidate area.

According to an exemplary embodiment, an endoscope system includes theimage processing apparatus described above, and the endoscope.

Advantages of the invention will be set forth in the description whichfollows, and in part will be obvious from the description, or may belearned by practice of the invention. Advantages of the invention may berealized and obtained by means of the instrumentalities and combinationsparticularly pointed out hereinafter.

BRIEF DESCRIPTION OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the general description given above and the detaileddescription of the embodiments given below, serve to explain theprinciples of the invention.

FIG. 1 is a block diagram schematically illustrating an example of aconfiguration of an endoscope system according to embodiments.

FIG. 2 is a view for describing a mode of use of the endoscope systemaccording to the embodiments.

FIG. 3 is a flowchart schematically illustrating an example of an imageprocess performed by an image processing apparatus according to theembodiments.

FIG. 4 is a flowchart schematically illustrating an example of aninterference candidate area extracting process according to theembodiments.

FIG. 5 is a view for describing the image process according to theembodiments, that is, a schematic view illustrating an image acquired byan endoscope.

FIG. 6 is a view for describing the image process according to theembodiments, that is, a schematic view illustrating extractedinterference candidate areas.

FIG. 7 is a view for describing the image process according to theembodiments, that is, a schematic view illustrating labeled interferencecandidate areas.

FIG. 8 is a flowchart schematically illustrating an example of aninterference area selecting process according to a first embodiment.

FIG. 9 is a view for describing the image process according to theembodiments, that is, a schematic view illustrating a selectedinterference area.

FIG. 10A is a view for describing the image process according to theembodiments, that is, an example of an image before a correctingprocess.

FIG. 10B is a view for describing the image process according to theembodiments, that is, an example of the image after the correctingprocess.

FIG. 11A is a view for describing a non-corresponding area, that is, aschematic view illustrating a left image.

FIG. 11B is a view for describing a non-corresponding area, that is, aschematic view illustrating a right image.

FIG. 12A is a flowchart schematically illustrating an example of aninterference area selecting process according to a second embodiment.

FIG. 12B is a flowchart schematically illustrating an example of theinterference area selecting process according to the second embodiment.

DETAILED DESCRIPTION First Embodiment

The first embodiment of the invention is described. The embodimentrelates to a three-dimensional (3D) endoscope system. In the endoscopesystem according to the embodiment, a specific image processing isperformed on a video acquired by capturing. In the image processing, animage area that gives some kind of visual discomfort to a user isdetected, and correction for reducing the discomfort is performed on thedetected image area.

The 3D endoscope includes a stereo camera having a pair of right andleft optical systems and an imaging device and captures a pair of rightand left subject images. The left image captured by the left opticalsystem and the left imaging device and the right image captured by theright optical system and the right imaging device are obtained as thesubject images with disparity approximate to that in a case of thebinocular vision of a human. The endoscope system displays an imagebased on the left image and the right image on the three-dimensionaldisplay apparatus, thereby, allowing the user to recognize athree-dimensional image of a subject.

<Configuration of Endoscope System>

An example of a configuration of an endoscope system 1 according to theembodiment is illustrated in a block diagram of FIG. 1. The endoscopesystem 1 includes an image processing apparatus 100, an endoscope 200, adisplay 300, and a light source apparatus 350. In the embodiment, theendoscope 200 is described as the 3D endoscope; however, the endoscope200 is not limited thereto. A part of the endoscope 200 is inserted intoan inside of a subject, thereby, capturing an image of the inside of thesubject, and the endoscope generates image data of the inside of thesubject. The image processing apparatus 100 performs various kinds ofimage processes or the like on the image data generated by the endoscope200. The display 300 displays an image processed by the image processingapparatus 100. The light source apparatus 350 is a light source ofillumination light for illuminating a range of which an image iscaptured by the endoscope 200.

For example, the endoscope 200 is a rigid endoscope for surgery and hasan insertion part having an elongated shape, the insertion partconfigured to be inserted into the subject. The endoscope 200 isconnected to the image processing apparatus 100 and the light sourceapparatus 350 via a cable. For example, the insertion part may have aconfiguration in which a distal portion of the insertion part has abending portion, which is actively bent, and is bent in a direction thatthe user wants through an operation by the user.

The distal portion of the insertion part has an imaging unit 210, animaging optical system 220, and an illumination unit 230. The imagingunit 210 and the imaging optical system 220 both have two right and leftsystems for acquiring a 3D image. The imaging unit 210 has a left imageacquisition unit 211 and a right image acquisition unit 212 that haverespective imaging devices such as a CCD image sensor, for example. Theimaging optical system 220 includes a left optical system 221, whichforms a subject image to be in focus on an imaging plane of the imagingdevice of the left image acquisition unit 211, and a right opticalsystem 222, which forms a subject image to be in focus on an imagingplane of the imaging device of the right image acquisition unit 212. Theillumination unit 230 includes an optical system that emits in adirection toward a subject illumination light that is emitted from thelight source apparatus 350 and is guided by an optical fiber.

An image of the subject illuminated by the illumination light emittedfrom the illumination unit 230 is in focus on the imaging plane of theeach imaging device of the imaging unit 210 via the imaging opticalsystem 220. The imaging unit 210 generates image data based on thesubject image by an imaging operation. The image data contains a leftimage that is generated by the left image acquisition unit 211 and aright image that is generated by the right image acquisition unit 212.The image data is transmitted to the image processing apparatus 100 viathe cable.

The image processing apparatus 100 includes an image acquisition circuit110, an image memory 120, an interference area setting circuit 130, aninterference area corrector 140, and a display image generator 150.

The image acquisition circuit 110 acquires the image data from theendoscope 200. The image memory 120 divides the image data acquired bythe image acquisition circuit 110 into left image data and right imagedata and stores both the image data. For example, the image memory 120includes a semiconductor memory such as a DRAM. Various processes areperformed by using the image stored in the image memory 120.

The interference area setting circuit 130 sets an interference area ineach of the left image data and the right image data stored in the imagememory 120. Here, the interference area is an image area that givesdiscomfort to a user in observation. The interference area settingcircuit 130 functions as an interference candidate area extractor 131, alabeling circuit 132, and an interference area selector 133. Theinterference candidate area extractor 131 extracts a candidate of theinterference area as an interference candidate area based on acharacteristic of the image acquired from the image memory 120. Thelabeling circuit 132 labels each of one or a plurality of extractedinterference candidate areas. The interference area selector 133 removesan area such as a bright spot other than the interference area from thelabeled interference candidate areas acquired from the labeling circuit132, based on the characteristic of the image acquired from the imagememory 120, and selects an interference area.

The interference area corrector 140 acquires the image from the imagememory 120, acquires data of the interference area from the interferencearea selector 133, and performs correction on the set interference areain the image. The correction is an image process for reducing thediscomfort of the user with the interference area. For example, thecorrection can include a process of reducing luminance of theinterference area. The display image generator 150 performs anotherimage process on image data corrected by the interference area corrector140. The image process includes various processes for making the imagesuitable for a display on the display 300 and also includes an imageprocess for displaying a three-dimensional image, for example. Thedisplaying image data processed by the display image generator 150 isoutput to the display 300.

For example, the interference area setting circuit 130, the interferencearea corrector 140, the display image generator 150, and the likeinclude an integrated circuit or the like such as an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), a graphics processing unit (GPU), or a central processing unit(CPU). The interference area setting circuit 130, the interference areacorrector 140, the display image generator 150, and the like may be eachconfigured of one integrated circuit or the like or may be eachconfigured of a combination of a plurality of integrated circuits. Inaddition, two or more of the interference area setting circuit 130, theinterference area corrector 140, the display image generator 150, andthe like may be each configured of one integrated circuit or the like.The integrated circuits operate in accordance with a program recorded ina recording device (not illustrated) provided in the image processingapparatus 100 or recorded in a recording area in the integrated circuit.

The display 300 is a 3D display. The display 300 displays athree-dimensional image, based on the displaying image data acquiredfrom the image processing apparatus 100. For example, the display 300 isthe 3D display using polarized light. In this case, the user wearspolarized glasses to see the image displayed on the display 300,thereby, being able to recognize the displayed image as thethree-dimensional image.

<Mode of Use of Endoscope System>

A mode of use of the endoscope system 1 is described. FIG. 2 is aschematic view illustrating an example of a state of a procedure usingthe endoscope system 1. An insertion part 240 of the endoscope 200 isinserted into an inside of a subject 420. As illustrated in FIG. 2, adistal portion of the insertion part 240 has the left optical system 221and the right optical system 222 of the imaging optical system 220. Thethree-dimensional image is displayed on the display 300, based on theleft image and the right image that are acquired via the two opticalsystems. In addition, the distal portion of the insertion part 240 hasthe illumination unit 230. Two illumination windows, through which theillumination light is emitted, are drawn on the illumination unit 230 inFIG. 2; however, one or three or more illumination windows may beprovided. In addition, the left optical system 221, the right opticalsystem 222, and the illumination unit 230 may have any positionalrelationship or the like. For example, the illumination windows of theillumination unit 230 may be arranged in a ring shape so as to surroundthe left optical system 221 and the right optical system 222.

In addition to the insertion part 240 of the endoscope 200, forceps 410or the like is inserted into the inside of the subject 420, and aprocedure is performed on tissue 422 which is a procedure target. Onepair of forceps 410 is drawn in FIG. 2; however, a plurality of pairs offorceps may be inserted into the subject 420, or a procedure tool thatperforms a procedure on the tissue 422 by using high-frequency electricpower, ultrasonic vibration, heat energy, or the like may be insertedinto the inside of the subject 420. While the user views the image thatis acquired by the endoscope 200 and displayed on the display 300, theuser performs the procedure on the tissue 422 which is the proceduretarget by using the forceps 410 or the like.

<Operation of Image Processing Apparatus>

An image processing that is performed by the image processing apparatusis described with reference to a flowchart illustrated in FIG. 3.

In Step S101, the image acquisition circuit 110 acquires the image dataobtained by imaging from the imaging unit 210 of the endoscope 200.Here, the acquired image data contains left image data obtained by theleft image acquisition unit 211 and right image data obtained by theright image acquisition unit 212. In Step S102, the image memory 120temporarily stores the acquired image data. In this case, the left imagedata and the right image data are separately stored.

In Step S103, the interference area setting circuit 130 selects an imageon which the following processes are to be performed. Specifically, theinterference area setting circuit 130 selects one from the left imagedata and the right image data.

In Step S104, the interference candidate area extractor 131 performs aninterference candidate area extracting process. In the interferencecandidate area extracting process, the interference candidate areaextractor 131 extracts, as an interference candidate area, an area otherthan a region having the characteristic of the image such as acharacteristic of a color of a living body, in the image data stored inthe image memory 120. For example, the interference candidate areaextractor 131 extracts the interference candidate area by using thecharacteristic related to color such as chroma information or hueinformation of the image. An example of the interference candidate areaextracting process is described with reference to FIG. 4.

In Step S201, the interference candidate area extractor 131 selects anarea which becomes a target of processes of subsequent Step S202 to StepS207. The selected area is a rectangular area having one or more pixels,for example. Incidentally, the selected area may not have a rectangularshape. In Step S202, the interference candidate area extractor 131calculates chroma of the area which is a target selected in Step S201.For example, the interference candidate area extractor 131 converts RGBsignals into YCbCr signals and calculates the chroma. Of the YCbCrsignals, the Cb signal and Cr signal are normalized into the Y signal.In this manner, a reflection intensity of the subject isSat=(Σ_(y=1) ^(SizeY)Σ_(x=1) ^(SizeX)NormCb(x,y))²+(Σ_(y=1)^(SizeY)Σ_(x=1) ^(SizeX)NormCr(x,y))²  (1)removed, and only the characteristic of the color can be handled. Forexample, a characteristic value of the chroma can be calculated, basedon a distance from an origin to a point indicated by a Cb signal valueand a Cr signal value normalized by the Y signal in a space with axes ofCb and Cr normalized by the Y signal. Specifically, the feature value ofthe chroma can be calculated from the following Equation (1), forexample: Here, Sat represents chroma within an area, x and y represent apixel position, SizeX and SizeY represent a size of the area set in StepS201, NormCb and NormCr represent a Cb signal value and a Cr signalvalue normalized by the Y signal.

Incidentally, an example in which a YCbCr space is used in thecalculation of chroma is described here; however, an HLS space or thelike may be used.

In Step S203, the interference candidate area extractor 131 determineswhether or not the chroma calculated in Step S202 is equivalent to thecharacteristic of the living body. Here, a characteristic of a livingbody for comparison may be set in advance. For example, when the chromais a value of an achromatic color or is approximate thereto, the chromais determined not to be equivalent to the characteristic of the livingbody. When the chroma is not equivalent to the characteristic of theliving body, the process proceeds to Step S206 to be described below. Onthe one hand, when the chroma is equivalent to the characteristic of theliving body, the process proceeds to Step S204.

In Step S204, the interference candidate area extractor 131 calculateshue of the area which is the target. Similarly to the calculation of thechroma, the calculation of the hue may be performed by using the Cbsignal value and the Cr signal value normalized by the Y signal. Forexample, a characteristic value of the hue can be calculated, based onan angle of the point indicated by the Cb signal value and the Cr signalvalue normalized by the Y signal in a space with axes of Cb and Crnormalized by the Y signal. Specifically, the feature value of the huecan be calculated from the followingH=arctan(Σ_(y=1) ^(SizeY)Σ_(x=1) ^(SizeX)NormCb(x,y),Σ_(y=1)^(SizeY)Σ_(x=1) ^(SizeX)NormCr(x,y))  (2)Equation (2), for example:Here, H represents hue, and arctan(a, b) represents an arctangent valueof a and b. Instead of arctangent, a sign and a slope in a Cb and Crspace normalized by the Y signal may be calculated.

Incidentally, an example in which a YCbCr space is used in thecalculation of hue is described here;

however, the HLS space or the like may be used.

In Step S205, the interference candidate area extractor 131 determineswhether or not the hue calculated in Step S204 is equivalent to thecharacteristic of the living body. A characteristic of a living body forcomparison may be set in advance. For example, when the hue is reddish,the hue is determined to be equivalent to the characteristic of theliving body. When the hue is not equivalent to the characteristic of theliving body, the process proceeds to Step S206. On the one hand, whenthe hue is equivalent to the characteristic of the living body, theprocess proceeds to Step S207.

In Step S206, the interference candidate area extractor 131 extracts thearea as an interference candidate area. Then, the process proceeds toStep S207. As described above, when the chroma is not equivalent to thecharacteristic of the living body or when the chroma is equivalent tothe characteristic of the living body but the hue is not equivalent tothe characteristic of the living body, the area is extracted as theinterference candidate area. Otherwise, the area is not extracted as theinterference candidate area.

In Step S207, the interference candidate area extractor 131 determineswhether or not every area in the image is processed as the target area.When every area is not processed as the target area, the process returnsto Step S201. On the one hand, when every area is processed as thetarget area, the interference candidate area extracting process isended, and the process returns to the image process.

The interference candidate area extracting process is described withreference to a schematic view. For example, a case where an image 710 asillustrated in FIG. 5 is acquired by the endoscope 200 is considered. Inthe image 710 illustrated in FIG. 5, an internal organ 711 which is theliving body and forceps 712 are shown. In general, the forceps 712 havea color characteristic different from that of the internal organ 711 orthe like which is an observation target. In addition, a region of theforceps 712 close to the imaging unit 210 of the endoscope 200 hashigher luminance than another portion thereof. In addition, in the image710, the living body such as the internal organ 711 other than theforceps 712 has a bright spots 716. The bright spot 716 has RGB signalvalues that are each approximate to a saturation value, and thus thechroma that is calculated in the process of Step S202 can be a value ofan achromatic color or can be approximate thereto. In this case, thebright spot 716 has the chroma that is not equivalent to thecharacteristic of the living body and can be extracted as theinterference candidate area. In addition, even when the chromacalculated in the process of Step S202 is determined to be equivalent tothe characteristic of the living body, the hue of the bright spot 716 isdifferent from that of the internal organ 711 or the like in some cases.In this case, the bright spot 716 has the hue that is not equivalent tothe characteristic of the living body and can be extracted as theinterference candidate area. As described above, when theabove-described interference candidate area extracting process isperformed on the image 710, for example, the forceps 712, the brightspots 716, and the like can be extracted as the interference candidatearea. FIG. 6 illustrates a schematic view of extracted interferencecandidate area data 720. In FIG. 6, white areas represent the extractedinterference candidate areas. That is, the interference candidate areadata 720 contains information indicating a first area 722 correspondingto the forceps 712 and second areas 726 corresponding to the brightspots 716.

Incidentally, an example in which the interference candidate area isextracted by using the chroma information and the hue information isprovided; however, the invention is not limited thereto. Theinterference candidate area may be extracted by using one of the chromainformation and the hue information. In addition, the interferencecandidate area may be extracted by using a characteristic related tocolor other than the chroma information or the hue information. Inaddition, the interference candidate area may be extracted by using acharacteristic of the image other than the color, the characteristicindicating a different value between the observation target and theinterference area.

The description back to FIG. 3 is continued. In the embodiment,correction for reducing luminance in the first area 722 corresponding tothe forceps 712 is performed, and correction is not performed on thesecond area 726 corresponding to the bright spot 716. Therefore, it isnecessary to distinguish between the first area 722 and the second area726 in the subsequent process.

In Step S105, the labeling circuit 132 executes a labeling process oflabeling the interference candidate areas extracted in the interferencecandidate area extracting process. In the labeling process, the labelingcircuit 132 first sets areas having one or more pixels in order. Whenthe set area is the interference candidate area, and a labeled area isnot present in eight adjacent areas around the area, the set area islabeled with new number. When the set area is the interference candidatearea, and a labeled area is present in eight adjacent areas around thearea, the set area is labeled with the same number as a label present inadvance. An area other than the interference candidate area is notlabeled. Such a method described here is an example of the labelingprocess, and labeling may be performed by any other method includingvarious methods which are generally used.

As an example, FIG. 7 illustrates a result of the labeling processperformed on the interference candidate area data illustrated in FIG. 6.As illustrated in FIG. 7, in labeling data 730, a label “1” is assignedto the first area 722 corresponding to the forceps 712, and labels “2”,“3”, and “4” are assigned to the second areas 726 corresponding to thebright spots 716.

In Step S106, the interference area selector 133 performs aninterference area selecting process of selecting an interference area,based on the image data stored in the image memory 120 and labeling dataprocessed by the labeling circuit 132. As illustrated in FIG. 2, ingeneral, the procedure tool such as the forceps 410 is inserted towardthe tissue 422 from an outer side of an angle of view in the imagingoptical system 220 of the insertion part 240 of the endoscope 200.Therefore, in FIG. 5, the forceps 712 are closest to the imaging opticalsystem 220 of the endoscope 200 at an image end, and thus there is ahigh possibility that it is not possible to achieve a stereoscopic view.In addition, the forceps 712 have high luminance because the forceps 712are closest to the illumination unit 230 at the image end, and thus theforceps 712 are likely to catch an eye. In the embodiment, an areaextended from a peripheral edge of the image 710 toward an inside of theimage 710 is set as the interference area from the interferencecandidate areas such that an area such as the forceps 712 which is closeto the endoscope 200 is extracted as the interference area. That is, theinterference area is selected, based on positional information relatedto the interference candidate area. The interference area selectingprocess is described with reference to FIG. 8.

In Step S301, the interference area selector 133 selects a label whichbecomes a target of processes of Step S302 to Step S306. For example, inan example illustrated in FIG. 7, labels 1 to 4 are selected in order.

In Step S302, the interference area selector 133 determines a minimumvalue minX of an x coordinate and a minimum value minY of a y coordinatefrom pixels within an area with the label which is the target. Forexample, minX represents the leftmost pixel in the area with thecorresponding label in the image. For example, minY represents theuppermost pixel in the area with the corresponding label in the image.

In Step S303, the interference area selector 133 determines whether ornot minX≠0 and minY≠0. When minX is 0, this indicates that the area withthe corresponding label is in contact with a left end of the image 710.When minY is 0, this indicates that the area with the correspondinglabel is in contact with an upper end of the image 710. When minX=0 orminY=0, the process proceeds to Step S306 to be described below. On theone hand, when minX≠0 and minY≠0, the process proceeds to Step S304.

In Step S304, the interference area selector 133 determines a maximumvalue maxX of the x coordinate and a maximum value maxY of the ycoordinate from the pixels within the area with the label which is thetarget. For example, maxX represents the rightmost pixel in the areawith the corresponding label in the image. For example, maxY representsthe lowermost pixel in the area with the corresponding label in theimage.

In Step S305, the interference area selector 133 determines whether ornot maxX≠width of image 710 and minY≠height of image 710. When maxX isthe width of the image 710, this indicates that the area with thecorresponding label is in contact with a right end of the image 710.When maxY is the height of the image 710, this indicates that the areawith the corresponding label is in contact with a lower end of the image710. When maxX=width of image 710 or maxY=height of image 710, theprocess proceeds to Step S306. On the one hand, when maxX≠width of image710 or minY≠height of image 710, the process proceeds to Step S307.

In Step S306, the interference area selector 133 selects, as aninterference area, the interference candidate area related to thecorresponding label. That is, when the interference candidate area is incontact with an end portion or a side of the image 710, the area isselected as the interference area. Then, the process proceeds to StepS307.

In Step S307, the interference area selector 133 determines whether ornot the processes described above are ended regarding every label. Whenthe processes are not ended on every label, the process returns to StepS301. On the one hand, when the processes are ended on every label, theinterference area selecting process is ended, and the process returns tothe image process.

FIG. 9 illustrates a schematic view of interference area data generatedin the interference area selecting process. Only an area 742corresponding to the forceps 712 is selected in interference area data740.

The description back to FIG. 3 is continued. In Step S107, theinterference area corrector 140 performs an interference area correctingprocess. That is, the interference area corrector 140 acquires data ofan image as a process target from the image memory 120 and acquiresinterference area data indicating the interference area from theinterference area selector 133. The interference area corrector 140determines a pixel corresponding to the interference area in the imageand performs a correcting process of reducing a luminance value of theimage on the corresponding pixel.

A reduction in luminance value may be stored in a table or the like ormay be calculated depending on the luminance value. For example, it ispreferable to reduce the luminance to two thirds of a maximum value ofthe luminance of a pixel in the interference area. For example,adjustment of the luminance value may be performed by γ correction.

When the luminance value is reduced in the interference area by thecorrecting process, a color may be intensely perceived in a case or thelike where a living body is reflected in the forceps 712, for example.Hence, a correcting process of reducing chroma proportional to areduction in the luminance value may also be performed on a pixelrelated to the interference area, for example. As described above, thecorrecting process of making the interference area inconspicuous in theimage is performed. As a result, an interference feeling occurring whenviewing the image is reduced in a processed image.

In Step S108, the image processing apparatus 100 determines whether ornot the left image and the right image stored in the image memory 120are both processed. When both images are processed, the process proceedsto Step S109. Otherwise, the process returns to Step S103.

In Step S109, the display image generator 150 acquires image datasubjected to the correcting process, performs another image process fordisplay including an image process for the three-dimensional display,and generates a display image. The display image generator 150 outputsdata of the generated display image to the display 300 and causes thedisplay 300 to display the display image.

FIG. 10A illustrates an example of a first image 750 when the imageprocess according to the embodiment is not performed, and FIG. 10Billustrates an example of a second image 760 when the image process isperformed. Halation occurs in forceps 751 in the first image 750illustrated in FIG. 10A. By comparison, luminance of forceps 761 isreduced in the second image 760 illustrated in FIG. 10B. On the onehand, in the first image 750, a bright spot 756 shown in a living body755 remains as a bright spot 766 shown in a living body 765 also in thesecond image 760.

<Feature of Endoscope System>

In the endoscope system 1 according to the embodiment, the luminance ofthe interference area such as a high-luminance area of a procedure toolsuch as forceps that enters from an outside of a screen is reduced, forexample. As a result, the discomfort felt by the user is reduced.

MODIFICATION EXAMPLES

FIG. 11A illustrates a schematic view of a left image 770, and FIG. 11Billustrates a schematic view of a right image 780. In general, a visualfield is different between a pair of left image 770 and right image 780.Therefore, the left image 770 contains a first non-corresponding area779 that is captured in the left image 770 but is not captured in theright image 780. Similarly, the right image 780 contains a secondnon-corresponding area 789 that is captured in the right image 780 butis not captured in the left image 770.

The first non-corresponding area 779 and the second non-correspondingarea 789 are areas that cannot be seen in the stereoscopic view. Whenthe first non-corresponding area 779 and the second non-correspondingarea 789 are conspicuous, a user feels discomfort. Hence, theinterference area setting circuit 130 may set the firstnon-corresponding area 779 and the second non-corresponding area 789 asthe interference area, in addition to the interference area according tothe embodiment described above or instead of the interference areaaccording to the embodiment described above. That is, the interferencearea setting circuit 130 may set, on an image, an interference areaincluding the area captured only by one image of a pair of imagescaptured by using a three-dimensional endoscope. As a result, theinterference area corrector 140 performs, on the first non-correspondingarea 779 and the second non-corresponding area 789, the same imageprocess as the image process that is performed on the interference areain the embodiment described above. In this manner, a line of vision of auser is unlikely to be directed to the non-corresponding area, and thediscomfort felt by the user is reduced.

In addition, the interference area setting circuit 130 may set, as aninterference area, from a range of the interference area as selected inthe embodiment described above, a range being contained in thenon-corresponding area, and the interference area corrector 140 mayperform correction on the corresponding area.

In addition, in the embodiment described above, an example in which theentire forceps 712 are set as the interference area; however, theinvention is not limited thereto. For example, the interference areasetting circuit 130 may set, as the interference area, an area of theforceps 712 having a luminance value higher than a predetermined value.

As described above, the interference area in the embodiment describedabove is described as an example, and the interference area can bevarious areas that give discomfort or the like to a user.

Second Embodiment

The second embodiment is described. Here, differences from the firstembodiment are described, and the same reference signs are assigned tothe same portions, and thereby the description thereof is omitted. Inthe first embodiment, the interference area is determined, based on thefact that the interference area is in contact with the end portion orthe side of the image 710. However, in this method, there is a concernthat, when the bright spot 716 is present in contact with the endportion of the image 710, the bright spot 716 is extracted as theinterference area. Hence, in the embodiment, the interference area isdetermined as follows, such that an area of the procedure tool is morestably selected. That is, the procedure tool often has a cylindricalhandle, in general, and thus the luminance is distributed in a broadrange in the handle of the procedure tool. On the one hand, asignificant bias in distribution of the luminance occurs in the brightspot of the living body. In the embodiment, the interference area isdetermined by using luminance distribution in a label and a size of thelabel. The interference area selecting process according to theembodiment is described with reference to a flowchart illustrated inFIGS. 12A and 12B.

In the interference area selecting process according to the embodiment,processes of Step S401 to Step S406 are the same as those of Step S301to Step S306 in the interference area selecting process according to thefirst embodiment. To put it briefly, In Step S401, the interference areaselector 133 selects a label which becomes a target of the subsequentprocesses. In Step S402, the interference area selector 133 determines aminimum value minX of the x coordinate and a minimum value minY of the ycoordinate from pixels within an area with the label which is thetarget. In Step S403, the interference area selector 133 determineswhether or not minX≠0 and minY≠0. When minX=0 or minY=0, the processproceeds to Step S406 to be described below. On the one hand, whenminX≠0 and minY≠0, the process proceeds to Step S404. In Step S404, theinterference area selector 133 determines a maximum value maxX of the xcoordinate and a maximum value maxY of the y coordinate from the pixelswithin the area with the label which is the target. In Step S405, theinterference area selector 133 determines whether or not maxX≠width ofimage 710 and minY≠height of image 710. When maxX=width of image 710 ormaxY=height of image 710, the process proceeds to Step S406. On the onehand, when maxX≠width of image 710 or minY≠height of image 710, theprocess proceeds to Step S413. In Step S406, the interference areaselector 133 selects, as an interference area, an interference candidatearea related to the corresponding label. Then, the process proceeds toStep S407. That is, when the interference candidate area is in contactwith an end of the image 710, the corresponding area is selected as theinterference area, and the process proceeds to Step S407.

Processes of Step S407 to Step 409 are processes of extracting an areaindicating a bright spot 716 of the living body by using the luminancedistribution in the label. Many areas having a saturated luminance valueare present in areas of the bright spot 716 extracted as theinterference candidate area. On the one hand, the procedure tool oftenhas a cylindrical handle, in general, and thus distribution of luminancevalue is wide. Hence, the distribution of the luminance value isfocused, and the bright spot 716 of the living body is removed from theinterference area.

In Step S407, the interference area selector 133 calculates a luminancevalue in each of the areas in the label. Here, the areas are rectangularareas having one or more pixels, for example. Incidentally, the area maynot have a rectangular shape. In addition, the luminance value may be aluminance signal value that is generated based on the RGB signals, forexample.

In Step S408, the interference area selector 133 calculates anoccupation percentage of areas having a maximum luminance value in thelabel. Here, the occupation percentage of the areas having the maximumluminance value in the label is a value obtained by dividing the numberof areas having the maximum luminance value or luminance values within apredetermined range, compared with the maximum luminance value, by thenumber of areas, in the label. Instead of the occupation percentage ofthe areas having the maximum luminance value or the luminance valueswithin the predetermined range, compared with the maximum luminancevalue, in the label, an occupation percentage of areas having asaturation value of the luminance or luminance values within apredetermined range, compared with the saturation value, in the label,may be used. In addition, an occupation percentage of areas havinganother predetermined luminance value in the label may be used. Forexample, a predetermined threshold value may be determined based on ahistogram of the luminance, and an occupation percentage of areas havingluminance equal to or higher than the threshold value in the label maybe used.

In Step S409, the interference area selector 133 performs thresholdvalue determination on the calculated occupation percentage of areashaving the maximum luminance value in the label. That is, theinterference area selector 133 determines whether or not the percentageis equal to or lower than the predetermined threshold value. Thethreshold value used here is not limited thereto and is preferably about70%, for example. When the percentage of the areas having the maximumluminance value in the label is higher than the threshold value, theprocess proceeds to Step S412, and the corresponding areas are removedfrom the interference area. On the one hand, when the percentage isequal to or lower than the threshold value, the process proceeds to StepS410.

Processes of Step S410 and Step S411 are processes of extracting an areaindicating the bright spot 716 of the living body by using a size of thelabel. The bright spot 716 extracted as the interference candidate areaincludes a very small bright spot. In the very small bright spot 716,the percentage of the area having the maximum luminance value in thelabel is low, and thus the bright spot 716 of the living body is notaccurately selected based on the luminance distribution. Hence, the sizeof the label is focused, and the very small bright spot 716 of theliving body is removed from the interference area.

In Step S410, the interference area selector 133 calculates the numberof pixels in the same label. In Step S411, the interference areaselector 133 determines whether or not the number of pixels in the samelabel is equal to or smaller than a predetermined threshold value. Here,the threshold value is not limited thereto, for example, and may beabout 100 pixels with respect to a hi-vision image. When the number ofpixels in the same label is equal to or smaller than the predeterminedthreshold value, the process proceeds to Step S412. In Step S412, theinterference area selector 133 removes, from the interference area,areas of the label determined as the bright spot 716 in thedetermination described above from a target selected as the interferencearea and determines that the corresponding area is not the interferencearea. Then, the process proceeds to Step S413.

When the number of pixels is determined to be not equal to or smallerthan the threshold value in Step S411, the process proceeds to StepS413. That is, a label that is selected as the interference area and isset as a target is not removed from the interference area and isdetermined to be the interference area.

In Step S413, the interference area selector 133 determines whether ornot the processes described above are ended regarding every label. Whenthe processes are not ended on every label, the process returns to StepS401. On the one hand, when the processes are ended on every label, theinterference area selecting process is ended, and the process returns tothe image process.

In the interference area selecting process according to the embodiment,the interference area can be selected with higher accuracy, comparedwith the interference area selecting process according to the firstembodiment.

Incidentally, it is needless to say that the embodiment can be used tobe combined with the modification examples described above.

MODIFICATION EXAMPLES

In the two embodiments described above, the endoscope 200 is describedto be the 3D endoscope; however, a technology according to theembodiment is not limited to the 3D endoscope and can be similarlywidely applied to a two-dimensional endoscope including one imagingsystem.

In addition, In the two embodiments and the modification examplesdescribed above, an example of a case where the endoscope 200 is amedical rigid endoscope is described; however, the endoscope 200 can besimilarly applied to a medical soft endoscope. In addition, theendoscope 200 is not limited to the medical endoscope and can also beapplied to an industrial endoscope. In this case, a technology accordingto the embodiments and the modification examples can be used in a statein which an observation target and an instrument used with theindustrial endoscope are distinguished based on a characteristic such ascolor in an image.

As described above, of the technology described in the embodiments,control mainly described by the flowchart can be realized by using aprogram. The program can be installed in a circuit such as the FPGA. Inaddition, in a case of an operation in the CPU or the like, the programmay be stored in a recording medium or a recording unit, for example.Various methods may be used to perform recording to the recording mediumor the recording unit. The recording may be performed at the time ofproduct delivery, or the recording may be performed by using adistributed recording medium or using download via internet.

In addition, regarding methods of various kinds of determination ofextracting the interference candidate area, selecting the interferencearea, or the like in the image process, for example, the embodimentsdescribed above are provided as examples, and various methods can beemployed for such determination. Here, in the methods of various kindsof determination, the determination is not limited to simpledetermination based on the threshold value or the like as in theembodiments described above, and determination based on artificialintelligence or the like built on machine learning such as deep learningmay be employed, for example.

In addition, the image processing apparatus according to the embodimentsis not limited to the endoscope and can also be applied to a microscope,industrial equipment for inspection or the like, various medicalobservation apparatuses or the like.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details, and representative devices shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An image processing apparatus for athree-dimensional endoscope, the image processing apparatus comprising:an image acquisition circuit configured to acquire a pair of imagescaptured by using the endoscope; an interference area setting circuitconfigured to set, on an image among the pair of images, an interferencearea which is different from an observation target in the image withrespect to a first characteristic, the first characteristic being acharacteristic relating to color; and an interference area correctorconfigured to perform correction on the image, the correction comprising(i) specifying pixels in the image corresponding to the interferencearea as processing targets, and (ii) performing image processing toreduce luminance values of the pixels in the image specified as theprocessing targets, wherein the interference area setting circuitincludes: an interference candidate area extractor configured to extractat least one interference candidate area based on the firstcharacteristic, and an interference area selector configured to selectthe interference area from the at least one interference candidate areabased on positional information of the at least one interferencecandidate area in the image and an occupation percentage of areas havingpixels with luminance values equal to or higher than a threshold valuein the at least one interference candidate area, and wherein theinterference area selector is configured to select, as the interferencearea, an interference candidate area that is in contact with a side ofthe image and that extends from a peripheral edge of the image towardsan inside of the image.
 2. The image processing apparatus according toclaim 1, wherein the interference candidate area extractor is configuredto use chroma information as the first characteristic.
 3. The imageprocessing apparatus according to claim 1, wherein the interferencecandidate area extractor is configured to use hue information as thefirst characteristic.
 4. The image processing apparatus according toclaim 1, wherein the interference area selector is configured to selectthe interference area from the at least one interference candidate areabased on a size of the at least one interference candidate area.
 5. Theimage processing apparatus according to claim 1, wherein the correctionfurther comprises performing image processing to reduce chroma of theinterference area.
 6. The image processing apparatus according to claim1, wherein the interference area corrector is configured to determinethe threshold value based on a histogram of luminance values of pixelsin the at least one interference candidate area.
 7. The image processingapparatus according to claim 1, wherein the interference area settingcircuit is further configured to set, on each image of the pair ofimages, the interference area to include an area captured only on theimage of the pair of images.
 8. The image processing apparatus accordingto claim 7, wherein the correction further comprises performing imageprocessing to reduce chroma of the interference area.
 9. An endoscopesystem comprising: the image processing apparatus according to claim 1;and the endoscope.
 10. The image processing apparatus according to claim1, wherein the interference area comprises an area of a procedure tool.11. An image processing apparatus for a three-dimensional endoscope, theimage processing apparatus comprising: an image acquisition circuitconfigured to acquire a pair of images captured by using the endoscope;and a processor, wherein the processor is configured to: set, on animage among the pair of images, an interference area which is differentfrom an observation target in the image with respect to a firstcharacteristic, the first characteristic being a characteristic relatingto color, and the interference area being set by (i) extracting at leastone interference candidate area based on the first characteristic, and(ii) selecting the interference area from the at least one interferencecandidate area based on positional information of the at least oneinterference candidate area in the image and an occupation percentage ofareas having pixels with luminance values equal to or higher than athreshold value in the at least one interference candidate area, andperform correction on the image, the correction comprising (i)specifying pixels in the image corresponding to the interference area asprocessing targets, and (ii) performing image processing to reduceluminance values of the pixels in the image specified as the processingtargets, and wherein the selecting the interference area comprisesselecting, as the interference area, an interference candidate area thatis in contact with a side of the image and that extends from aperipheral edge of the image towards an inside of the image.